System and method for automatically detecting key behaviors by vehicles

ABSTRACT

Aspects of the disclosure relate generally to detecting discrete actions by traveling vehicles. The features described improve the safety, use, driver experience, and performance of autonomously controlled vehicles by performing a behavior analysis on mobile objects in the vicinity of an autonomous vehicle. Specifically, an autonomous vehicle is capable of detecting and tracking nearby vehicles and is able to determine when these nearby vehicles have performed actions of interest by comparing their tracked movements with map data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation reissue of application Ser.No. 15/847,390 filed Dec. 19, 2017 (now abandoned), which is anapplication for reissue of U.S. Pat. No. 9,216,737 (“the '737 patent”).The '737 patent is a continuation of U.S. patent application Ser. No.14/190,621, filed on Feb. 26, 2014 and issued as U.S. Pat. No.8,935,034, which is a continuation of U.S. patent application Ser. No.13/446,518, filed on Apr. 13, 2012, now issued as U.S. Pat. No.8,700,251, issued on Apr. 15, 2014, the disclosures of which areincorporated herein by reference, More than one reissue application hasbeen filed for the reissue of U.S. Pat. No. 9,216,737. The reissueapplications are application Ser. No. 17/138,339 (the presentapplication), application Ser. No. 15/847,390 filed Dec. 19, 2017;application Ser. No. 15/847,064 filed Dec. 19, 2017 (now abandoned); andapplication Ser. No. 17/138,281, filed concurrently herewith, each ofwhich is a reissue application of U.S. Pat. No. 9,216,737.

BACKGROUND

Autonomous vehicles use various computing systems to aid in thetransport passengers from one location to another. Some autonomousvehicles may require some initial input or continuous input from anoperator, such as a pilot, driver, or passenger. Other systems, forexample autopilot systems, may be used only when the system has beenengaged, which permits the operator to switch from a manual mode (wherethe operator exercises a high degree of control over the movement of thevehicle) to an autonomous mode (where the vehicle essentially drivesitself) to modes that lie somewhere in between.

BRIEF SUMMARY

In various aspects, the disclosure provides a vehicle having a steeringdevice (e.g., wheels that turn in the case of an automobile and a rudderin the case of a boat) and engine. The steering device may be controlledby a first user input controller (e.g., a steering wheel in the cockpitof a car), the engine may be controlled by a second user inputcontroller (e.g., accelerator in the case of a car or a throttle in thecase of boat), and both the engine and device may be controlled by aprocessor capable of executing computer instructions. The vehicleincludes one or more sensors (e.g., cameras, radar, laser range finders)for capturing information relating to the environment in which thevehicle is operating. The processor receives data from the sensors and,based in part on data from the sensors or received from external sourcesor both, issues a navigation command, where the navigation commandcomprises a command to the steering device relating to the intendeddirection of the vehicle (e.g., a command to turn the front wheels of acar 10 degrees to the left) or to the engine relating to the intendedvelocity of the vehicle (e.g., a command to accelerate). Navigationcommands may also include commands to brakes to slow the vehicle down,as well as other commands affecting the movement of the vehicle.

In one aspect, sensors are used to detect one or more nearby vehiclessurrounding the autonomous vehicle, and data corresponding to thesevehicles are sent to a processor. The processor analyzes the datacorresponding to the nearby vehicles by comparing the data with map datacontaining numerous road graph elements. The processor may thenassociate the nearby vehicles with a specific road graph element, anddetermine when a nearby vehicle has performed an action of interest. Theautonomously controlled vehicle may then alter its control strategybased on the occurrence of the action of interest. The road graphelements may include lanes of a roadway, and the action of interest mayinclude a nearby vehicle changing from a first lane to a second lane.

In another aspect, altering the control strategy includes positioningthe autonomous vehicle relative to the nearby vehicle in a predefinedmanner based on the occurrence of the action of interest. The controlstrategy may also be further altered based on actions of interest beingperformed by other nearby vehicles. The autonomous vehicle may alsoreceive a request to navigate between a first location and a secondlocation, where the control strategy includes having the autonomousvehicle travel along a path between the first and second locations.

In another aspect, the autonomous vehicle may filter data collectedregarding the position and movement of nearby vehicles. In particular,the data may be filtered to only include instances where a nearbyvehicle has performed an action of interest. As provided below, thisdetermination is made by comparing data for the nearby vehicle with mapdata accessed by the autonomous vehicle.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a functional diagram of an autonomous navigation system inaccordance with an aspect of the disclosure.

FIG. 2 is an exemplary design of the interior of an autonomous vehiclein accordance with an aspect of the disclosure.

FIG. 3 is a view of the exterior of an exemplary vehicle in accordancewith an aspect of the disclosure.

FIGS. 4A-4D are views of the sensor fields for an autonomous vehicle.

FIG. 5 is a view of an autonomous vehicle traveling autonomously inproximity to other vehicles.

FIG. 6 is a map image in accordance with an aspect of the disclosure.

FIG. 7 is a flow chart in accordance with an aspect of the disclosure.

FIG. 8 is a functional diagram of a communication system in accordancewith an aspect of the disclosure.

DETAILED DESCRIPTION

Aspects of the disclosure relate generally to detecting instances when avehicle has performed a discrete action of interest. In particular, adevice implementing the disclosed system is capable of detectingsurrounding vehicles using one or more sensors. The device may thendetermine when the surrounding vehicles have performed one of severalpredefined actions by comparing the sensor data with stored road graphdata. The system described below may be implemented as part ofautonomous driving vehicle. In turn, the autonomous vehicle may react tothe behavior of nearby objects in a way that decreases the likelihood ofan accident and increases the efficiency of travel.

As shown in FIG. 1 , an autonomous driving system 100 in accordance withone aspect of the invention includes a vehicle 101 with variouscomponents. While certain aspects of the invention are particularlyuseful in connection with specific types of vehicles, the vehicle may beany type of vehicle including, but not limited to, cars, trucks,motorcycles, buses, boats, airplanes, helicopters, lawnmowers,recreational vehicles, amusement park vehicles, trams, golf carts,trains, and trolleys. The vehicle may have one or more computers, suchas computer 110 containing a processor 120, memory 130 and othercomponents typically present in general purpose computers.

The memory 130 stores information accessible by processor 120, includinginstructions 132 and data 134 that may be executed or otherwise used bythe processor 120. The memory 130 may be of any type capable of storinginformation accessible by the processor, including a computer-readablemedium, or other medium that stores data that may be read with the aidof an electronic device, such as a hard-drive, memory card, ROM. RAM,DVD or other optical disks, as well as other write-capable and read-onlymemories. Systems and methods may include different combinations of theforegoing, whereby different portions of the instructions and data arestored on different types of media.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computer codeon the computer-readable medium. In that regard, the terms“instructions” and “programs” may be used interchangeably herein. Theinstructions may be stored in object code format for direct processingby the processor, or in any other computer language including scripts orcollections of independent source code modules that are interpreted ondemand or compiled in advance. Functions, methods and routines of theinstructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. For instance, although the systemand method is not limited by any particular data structure, the data maybe stored in computer registers, in a relational database as a tablehaving a plurality of different fields and records, XML documents orflat files. The data may also be formatted in any computer-readableformat. By further way of example only, image data may be stored asbitmaps comprised of grids of pixels that are stored in accordance withformats that are compressed or uncompressed, lossless (e.g., BMP) orlossy (e.g., JPEG), and bitmap or vector-based (e.g., SVG), as well ascomputer instructions for drawing graphics. The data may comprise anyinformation sufficient to identify the relevant information, such asnumbers, descriptive text, proprietary codes, references to data storedin other areas of the same memory or different memories (including othernetwork locations) or information that is used by a function tocalculate the relevant data.

The processor 120 may be any conventional processor, such ascommercially available CPU's. Alternatively, the processor may be adedicated device such as an ASIC or FPGA. Although FIG. 1 functionallyillustrates the processor, memory, and other elements of computer 110 asbeing within the same block, it will be understood by those of ordinaryskill in the art that the processor and memory may actually comprisemultiple processors and memories that may or may not be stored withinthe same physical housing. For example, memory may be a hard drive orother storage media located in a housing different from that of computer110. Accordingly, references to a processor or computer will beunderstood to include references to a collection of processors orcomputers or memories that may or may not operate in parallel. Ratherthan using a single processor to perform the steps described herein someof the components such as steering components and decelerationcomponents may each have their own processor that only performscalculations related to the component's specific function.

In various of the aspects described herein, the processor may be locatedremote from the vehicle and communicate with the vehicle wirelessly. Inother aspects, some of the processes described herein are executed on aprocessor disposed within the vehicle and others by a remote processor,including taking the steps necessary to execute a single maneuver.

Computer 110 may include all of the components normally used inconnection with a computer, such as a central processing unit (CPU),memory 130 (e.g., RAM and internal hard drives) storing data 134 andinstructions such as a web browser, an electronic display 142 (e.g., amonitor having a screen, a small LCD touch-screen or any otherelectrical device that is operable to display information), user input(e.g., a mouse, keyboard, touch screen and/or microphone), as well asvarious sensors (e.g., a video camera) for gathering explicit (e.g., agesture) or implicit (e.g. “the person is asleep”) information about thestates and desires of a person.

The vehicle may also include a geographic position component 144 incommunication with computer 110 for determining the geographic locationof the device. For example, the position component may include a GPSreceiver to determine the device's latitude, longitude and/or altitudeposition. Other location systems such as laser-based localizationsystems, inertial-aided GPS, or camera-based localization may also beused to identify the location of the vehicle. The location of thevehicle may include an absolute geographical location, such as latitude,longitude, and altitude as well as relative location information, suchas location relative to other cars immediately around it which can oftenbe determined with less noise than absolute geographical location.

The device may also include other features in communication withcomputer 110, such as an accelerometer, gyroscope or anotherdirection/speed detection device 146 to determine the direction andspeed of the vehicle or changes thereto. By way of example only,acceleration device 146 may determine its pitch, yaw or roll (or changesthereto) relative to the direction of gravity or a plane perpendicularthereto. The device may also track increases or decreases in speed andthe direction of such changes. The device's provision of location andorientation data as set forth herein may be provided automatically tothe user, computer 110, other computers and combinations of theforegoing.

The computer 110 may control the direction and speed of the vehicle bycontrolling various components. By way of example, if the vehicle isoperating in a completely autonomous mode, computer 110 may cause thevehicle to accelerate (e.g., by increasing fuel or other energy providedto the engine), decelerate (e.g., by decreasing the fuel supplied to theengine or by applying brakes) and change direction (e.g., by turning thefront two wheels).

As shown in FIG. 2 , Vehicle 101 may be equipped with a display 225 fordisplaying information relating to the overall status of the vehicle,particular sensors, or computer 110 in particular. The display 225 mayinclude computer generated images of the vehicle's surroundingsincluding, for example, the status of the computer, the vehicle itself,roadways, intersections, as well as other objects and information.

Computer 110 may use visual or audible cues to indicate whether computer110 is obtaining valid data from the various sensors, whether thecomputer is partially or completely controlling the direction or speedof the car or both, whether there are any errors, etc. Vehicle 101 mayalso include a status indicating apparatus, such as status bar 230, toindicate the current status of vehicle 101. In the example of FIG. 2 ,status bar 230 displays “D” and “2 mph” indicating that the vehicle ispresently in drive mode and is moving at 2 miles per hour. In thatregard, the vehicle may display text on an electronic display,illuminate portions of vehicle 101, or provide various other types ofindications. In addition, the computer may also have external indicatorswhich indicate whether, at the moment, a human or an automated system isin control of the vehicle, that are readable by humans, other computers,or both.

In one example, computer 110 may be an autonomous driving computingsystem capable of communicating with various components of the vehicle.Returning to FIG. 1 , computer 110 may be in communication with thevehicle's conventional central processor 160 and may send and receiveinformation from the various systems of vehicle 101, for example thebraking 180, acceleration 182, signaling 184, and navigation 186 systemsin order to control the movement, speed, etc. of vehicle 101. Inaddition, when engaged, computer 110 may control some or all of thesefunctions of vehicle 101 and thus be fully or partially autonomous. Itwill be understood that although various systems and computer 110 areshown within vehicle 101, these elements may be external to vehicle 101or physically separated by large distances.

FIG. 2 depicts an exemplary design of the interior of an autonomousvehicle. The autonomous vehicle may include all of the features of anon-autonomous vehicle, for example: a steering apparatus, such assteering wheel 210; a navigation display apparatus, such as navigationdisplay 215; and a gear selector apparatus, such as gear shifter 220.

The vehicle may include components for detecting objects external to thevehicle such as other vehicles, obstacles in the roadway, trafficsignals, signs, trees, etc. The detection system may include lasers,sonar, radar, cameras or any other detection devices. For example, ifthe vehicle is a small passenger car, the car may include a lasermounted on the roof or other convenient location. In one aspect, thelaser may measure the distance between the vehicle and the objectsurfaces facing the vehicle by spinning on its axis and changing itspitch. The vehicle may also include various radar detection units, suchas those used for adaptive cruise control systems. The radar detectionunits may be located on the front and back of the car as well as oneither side of the front bumper. In another example, a variety ofcameras may be mounted on the car at distances from one another whichare known so that the parallax from the different images may be used tocompute the distance to various objects which are captured by 2 or morecameras. These sensors allow the vehicle to understand and potentiallyrespond to its environment in order to maximize safety for passengers aswell as objects or people in the environment.

Many of these sensors provide data that is processed by the computer inreal-time, that is, the sensors may continuously update their output toreflect the environment being sensed at or over a range of time, andcontinuously or as-demanded provide that updated output to the computerso that the computer can determine whether the vehicle's then-currentdirection or speed should be modified in response to the sensedenvironment.

FIG. 3 illustrates a particular embodiment for a small passenger vehicle301 that includes lasers 310 and 311, mounted on the front and top ofthe vehicle, respectively. Laser 310 may have a range of approximately150 meters, a thirty degree vertical field of view, and approximately athirty degree horizontal field of view. Laser 311 may have a range ofapproximately 50-80 meters, a thirty degree vertical field of view, anda 360 degree horizontal field of view. The lasers may provide thevehicle with range and intensity information which the computer may useto identify the location and distance of various objects. In one aspect,the lasers may measure the distance between the vehicle and the objectsurfaces facing the vehicle by spinning on its axis and changing itspitch.

The vehicle may also include various radar detection units, such asthose used for adaptive cruise control systems. The radar detectionunits may be located on the front and back of the car as well as oneither side of the front bumper. As shown in the example of FIG. 3 ,vehicle 301 includes radar detection units 320-323 located on the side(only one side being shown), front and rear of the vehicle. Each ofthese radar detection units may have a range of approximately 200 metersfor an approximately 18 degree field of view as well as a range ofapproximately 60 meters for an approximately 56 degree field of view.

In another example, a variety of cameras may be mounted on the vehicle.The cameras may be mounted at predetermined distances so that theparallax from the images of 2 or more cameras may be used to compute thedistance to various objects. As shown in FIG. 3 , vehicle 301 mayinclude 2 cameras 330-331 mounted under a windshield 340 near the rearview mirror (not shown). Camera 330 may include a range of approximately200 meters and an approximately 30 degree horizontal field of view,while camera 331 may include a range of approximately 100 meters and anapproximately 60 degree horizontal field of view.

Each sensor may be associated with a particular sensor field in whichthe sensor may be used to detect objects. FIG. 4A is a top-down view ofthe approximate sensor fields of the various sensors. FIG. 4B depictsthe approximate sensor fields 410 and 411 for lasers 310 and 311,respectively based on the fields of view for these sensors. For example,sensor field 410 includes an approximately 30 degree horizontal field ofview for approximately 150 meters, and sensor field 411 includes a 360degree horizontal field of view for approximately 80 meters.

FIG. 4D depicts the approximate sensor fields 420A-423B and for radardetection units 320-323, respectively, based on the fields of view forthese sensors. For example, radar detection unit 320 includes sensorfields 420A and 420B. Sensor field 420A includes an approximately 18degree horizontal field of view for approximately 200 meters, and sensorfield 420B includes an approximately 56 degree horizontal field of viewfor approximately 80 meters. Similarly, radar detection units 321-323include sensor fields 421A-423A and 421B-423B. Sensor fields 421A-423Ainclude an approximately 18 degree horizontal field of view forapproximately 200 meters, and sensor fields 421B-423B include anapproximately 56 degree horizontal field of view for approximately 80meters. Sensor fields 421A and 422A extend passed the edge of FIGS. 4Aand 4D.

FIG. 4C depicts the approximate sensor fields 430-431 cameras 330-331,respectively, based on the fields of view for these sensors. Forexample, sensor field 430 of camera 330 includes a field of view ofapproximately 30 degrees for approximately 200 meters, and sensor field431 of camera 430 includes a field of view of approximately 60 degreesfor approximately 100 meters.

In another example, an autonomous vehicle may include sonar devices,stereo cameras, a localization camera, a laser, and a radar detectionunit each with different fields of view. The sonar may have a horizontalfield of view of approximately 60 degrees for a maximum distance ofapproximately 6 meters. The stereo cameras may have an overlappingregion with a horizontal field of view of approximately 50 degrees, avertical field of view of approximately 10 degrees, and a maximumdistance of approximately 30 meters. The localization camera may have ahorizontal field of view of approximately 75 degrees, a vertical fieldof view of approximately 90 degrees and a maximum distance ofapproximately 10 meters. The laser may have a horizontal field of viewof approximately 360 degrees, a vertical field of view of approximately30 degrees, and a maximum distance of 100 meters. The radar may have ahorizontal field of view of 60 degrees for the near beam, 30 degrees forthe far beam, and a maximum distance of 200 meters.

The sensors described may be used to identify, track and predict themovements of pedestrians, bicycles, other vehicles, or objects in theroadway. For example, the sensors may provide the location and shapeinformation of objects surrounding the vehicle to computer 110, which inturn may identify the object as another vehicle. The object's currentmovement may be also be determined by the sensor (e.g., the component isa self-contained speed radar detector) or by the computer 110 based oninformation provided by the sensors (e.g., by comparing changes in theobject's position data over time).

The computer may change the vehicle's current path and speed based onthe presence of detected objects. For example, the vehicle mayautomatically slow down if its current speed is 50 mph and it detects,by using its cameras and using optical-character recognition, that itwill shortly pass a sign indicating that the speed limit is 35 mph. Yetfurther, if the computer determines that an object is obstructing theintended path of the vehicle, it may maneuver the vehicle around theobstruction.

In accordance with one aspect, the autonomous vehicle's computer system110 may identify when another detected vehicle has performed aparticular action of interest. FIG. 5 illustrates an example of vehicle101 detecting surrounding objects. Using the sensors described above,autonomous vehicle 101 is capable of detecting surrounding vehicles510-550. As indicated by directional arrows A1, B1, C, and D, vehicle510 and vehicle 520 are currently traveling along the center lane, whileautonomous vehicle 101 and vehicle 530 are traveling north in theright-hand lane. Upon detecting the surrounding vehicles, computersystem 110 may track and record the position and movement (e.g.velocity, heading, and acceleration) of each surrounding vehicle.

The position and movement data for the detected vehicles 510-550 may bestored in database 137 of the autonomous driving computer system, asshown in FIG. 1 . The autonomous vehicle may then use this data tomaneuver vehicle 101 in a way that avoids collisions with nearbyvehicles. In accordance with one embodiment, computer system 110 mayfilter the position and movement data, so as to identify a vehicle'sposition and movement during particular actions of interest.

For example, database 138 may include a set of actions or behaviors ofinterest, such as the vehicle changing lanes or routes, and instructions132 may allow for computer system 110 to identify when a detectedvehicle has performed one or more of the behaviors of interest. Inparticular, computer system 110 may access the recorded position andmovement stored in database 137, as well as a road graph of theenvironment stored in database 136. By combining both sets of data,computer system 110, may then determine when one or more of the keybehaviors have occurred.

Returning to FIG. 5 , vehicle 101 may record the position and movementdata of vehicle 510 and vehicle 520. In doing so, vehicle 101 can detectthat vehicle 520 has de-accelerated and is changing its heading in thedirection of arrow A2. Similarly, vehicle 101 detects that vehicle 510has changed its heading as provided by arrow B2. Vehicle 101 may thenaccess stored road graph data that represents vehicle the currentenvironment of vehicles 510 and 520, such as road graph 600 illustratedin FIG. 6 . Based on the position and movement of vehicles 510 and 520,relative to vehicle 101, the computer system of vehicle 101 mayassociate vehicles 510 and 520 with the center lane of City Highway. Thecomputer system of vehicle 101 may then compare the changed heading ofvehicle 520 with the stored road graph to determine that vehicle 520 isturning from City Highway to Through Street, at point 610. Likewise, thecomputer system compares the change in heading of vehicle 510 with map600 to determine that the vehicle has changed from the center lane tothe adjacent lane to the right, at point 620.

In this way, vehicle 101 may associate and track all surroundingvehicles with a particular road graph element, such as a lane of travelor intersection. For example, dashed line 630 in FIG. 6 represents thepath of travel of vehicle 510. Provided that vehicle 510 remains withinan area that can be detected by vehicle 101, this path may be recordedby vehicle 101's computer system. By combining the vehicle's movementdata with map 600, vehicle 101 identifies instances along vehicle 510'spath where it has performed one or more behaviors of interest. Forexample, the computer system of vehicle 101 may identify any change inroad element as a behavior of interest, such as vehicle 510 travelingfrom City Highway, to First Avenue, and then to River Road. In addition,vehicle 101 may identify when vehicle 510 has changed lanes on aparticular roadway, such as at points 620 and 650.

Vehicle 101 may also filter the data collected for vehicle 510 so thatit only contains instances where vehicle 510 has performed an action ofinterest. As provided by dotted line 630 on map 600, vehicle 510 changesit's heading around point 640, as it begins to travel from a north-westdirection to a more south-west direction. While vehicle 101 will collectdata regarding vehicle 510's change in heading, computer 110 will alsodetermine that the change in heading does not correspond to an action ofinterest, as vehicle 510 merely travels along the same road graphelement. Vehicle 101 may, in turn, exclude the data corresponding tovehicle 510's change in heading at point 640 as being recorded as anaction of interest.

In another embodiment, autonomous vehicle 101 may transport itself,passengers, and/or cargo between two locations by following a route. Forexample, a driver may input a destination and activate an autonomousmode of the vehicle. In response, the vehicle's computer 110 maycalculate a route using a map, its current location, and thedestination. Based on the route (or as part of the route generation),the vehicle may determine a control strategy for controlling the vehiclealong the route to the destination. In accordance with one embodiment,computer system 110 may control the autonomous vehicle 101 to takeparticular actions in response to the actions of the surrounding objectsthat have been identified as performing a behavior of interest. Forexample, by changing lanes as provided by arrow B2 of FIG. 5 , vehicle101 may identify that vehicle 510 has performed a behavior of interestand thereby reevaluate the current control strategy of vehicle 101.Specifically, computer system 110 may determine that vehicle 101 shoulddecelerate in order to avoid travelling in vehicle 510's blind-spot,given that vehicle 510 is now travelling in the adjacent lane.

As another example, vehicle 520 may come to a stop for a period of timebefore making the left-hand turn designated by arrow A2. Computer system110 may identify this action as a behavior of interest, depending onwhich road element vehicle 520 is travelling. Specifically, if vehicle520 is determined to be in a left-hand turn lane, vehicle 101 may notidentify vehicle 520 having stopped will not as a behavior of interest.However, if vehicle 520 was travelling one lane over to the right, thefact that it has stopped could indicate that there is a backup ahead.Accordingly, vehicle 101 may adjust its control strategy based on whichroad graph element (e.g., lane) a vehicle is currently travelling.

Flow diagram 700 of FIG. 7 provides an example by which vehicle 101 maybe autonomously controlled in response to determined behaviors ofinterest. As provided in Block 710, vehicle 101 implements thedetermined control strategy by traveling along a determined route. Whiletraveling in accordance with the control strategy, vehicle 101 detectsthe presence of numerous objects within one or more of the vehicle'ssensor fields (Block 715). Upon detecting the objects, the computer 110may classify the object based on the data received by vehicle 101'ssensors (Block 720). For example, the sensor data could be used toclassify objects as being a pedestrian, bicycle, or vehicle. Asdescribed above, the vehicle's computer 110 also uses the sensor data todetermine the object's current state, such as speed, heading, andacceleration. Upon determining the objects classification and currentstate, computer 110 may then access map data associated with thedetected objects' environment (Block 725).

For each vehicle that has been detected, it may be determined whetherthe vehicle can be associated with a particular road graph element(e.g., a road, lane of traffic, intersection, or other map element)contained in the map data (Block 730). For example, based on a detectedvehicle's state information, computer 110 may determine that thedetected vehicle is travelling within a particular lane of trafficrepresented in the road graph. Computer 110 may then track the detectedvehicles as they travel along the associated road graph element, (Block735) and may determine when one of the detected vehicles has performed abehavior of interest (Block 740). Based on determining that a detectedvehicle has performed an action of interest, such as a lane change,computer 110 may then alter the control strategy of autonomous vehicle101 (Block 745). Blocks 715 through 745 may then be repeated untilautonomous vehicle 101 has reached its destination or the autonomouscontrol has otherwise terminated (Block 750). In this way, vehicle 101may further alter the control strategy upon any of the detected vehiclesperforming an action of interest.

Vehicle 101 may include one or more user input devices that enable auser to provide information to the autonomous driving computer 110. Forexample, a user, such as a passenger, may input a destination (e.g., 123Oak Street) into the navigation system using touch screen 217 or buttoninputs 219. In another example, a user may input a destination byidentifying the destination. In that regard, the computer system mayextract the destination from a user's spoken command.

The various systems described above may be used by the computer tooperate the vehicle and maneuver from one location to another. Forexample, a user may enter destination information into the navigationsystem, either manually or audibly. The vehicle may determine itslocation to a few inches based on a combination of the GPS receiverdata, the sensor data, as well as the detailed map information. Inresponse, the navigation system may generate a route between the presentlocation of the vehicle and the destination.

When the driver is ready to relinquish some level of control to theautonomous driving computer, the user may activate the computer. Thecomputer may be activated, for example, by pressing a button or bymanipulating a lever such as gear shifter 220. Rather than takingcontrol immediately, the computer may scan the surroundings anddetermine whether there are any obstacles or objects in the immediatevicinity which may prohibit or reduce the ability of the vehicle toavoid a collision. In this regard, the computer may require that thedriver continue controlling the vehicle manually or with some level ofcontrol (such as the steering or acceleration) before entering into afully autonomous mode.

Once the vehicle is able to maneuver safely without the assistance ofthe driver, the vehicle may become fully autonomous and continue to thedestination. The driver may continue to assist the vehicle bycontrolling, for example, steering or whether the vehicle changes lanes,or the driver may take control of the vehicle immediately in the eventof an emergency.

The vehicle may continuously use the sensor data to identify objects,such as traffic signals, people, other vehicles, and other objects, inorder to maneuver the vehicle to the destination and reduce thelikelihood of a collision. The vehicle may use the map data to determinewhere traffic signals or other objects should appear and take actions,for example, by signaling turns or changing lanes. Once the vehicle hasarrived at the destination, the vehicle may provide audible or visualcues to the driver. For example, by displaying “You have arrived” on oneor more of the electronic displays.

The vehicle may be only partially autonomous. For example, the drivermay select to control one or more of the following: steering,acceleration, braking, and emergency braking.

The vehicle may also have one or more user interfaces that allow thedriver to reflect the driver's driving a style. For example, the vehiclemay include a dial which controls the level of risk or aggressivenesswith which a driver would like the computer to use when controlling thevehicle. For example, a more aggressive driver may want to change lanesmore often to pass cars, drive in the left lane on a highway, maneuverthe vehicle closer to the surrounding vehicles, and drive faster thanless aggressive drivers. A less aggressive driver may prefer for thevehicle to take more conservative actions, such as somewhat at or belowthe speed limit, avoiding congested highways, or avoiding populatedareas in order to increase the level of safety. By manipulating thedial, the thresholds used by the computer to calculate whether to passanother car, drive closer to other vehicles, increase speed and the likemay change. In other words, changing the dial may affect a number ofdifferent settings used by the computer during its decision makingprocesses. A driver may also be permitted, via the user interface 225,to change individual settings that relate to the driver's preferences.In one embodiment, insurance rates for the driver or vehicle may bebased on the style of the driving selected by the driver.

Aggressiveness settings may also be modified to reflect the type ofvehicle and its passengers and cargo. For example, if an autonomoustruck is transporting dangerous cargo (e.g., chemicals or flammableliquids), its aggressiveness settings may be less aggressive than a carcarrying a single driver—even if the aggressive dials of both such atruck and car are set to “high.” Moreover, trucks traveling across longdistances over narrow, unpaved, rugged or icy terrain or vehicles may beplaced in a more conservative mode in order reduce the likelihood of acollision or other incident.

In another example, the vehicle may include sport and non-sport modeswhich the user may select or deselect in order to change theaggressiveness of the ride. By way of example, while in “sport mode”,the vehicle may navigate through turns at the maximum speed that issafe, whereas in “non-sport mode”, the vehicle may navigate throughturns at the maximum speed which results in g-forces that are relativelyimperceptible by the passengers in the car.

The vehicle's characteristics may also be adjusted based on whether thedriver or the computer is in control of the vehicle. For example, when aperson is driving manually the suspension may be made fairly stiff sothat the person may “feel” the road and thus drive more responsively orcomfortably, while, when the computer is driving, the suspension may bemade such softer so as to save energy and make for a more comfortableride for passengers.

The driver may also select to have his or her vehicle communicate withother devices. As shown in FIG. 8 , vehicle 101 may communicate over anetwork 820 with devices such as a remote server 810, a personalcomputer 830, a mobile device 840, or another autonomous vehicle 802. Inaddition, vehicles, such as vehicle 101 and vehicle 102, may wirelesslytransmit information directly to nearby vehicles using radio, cellular,optical or other wireless signals. Alternatively, vehicles maycommunicate with each via nodes that are shared among multiple vehicles,e.g., by using cell towers to call other cars or transmit and sendinformation to other cars via the Internet. The transmitted informationbetween vehicles may include data describing the vehicle or thevehicle's environment. For example, upon determine that a detectedvehicle has performed a behavior of interest, vehicle 101 may transmitdata to remote server 810 that identifies the detected vehicle and itstate (e.g., position, speed, acceleration, and heading). Remote Server810 may then record the data provided by vehicle 101 regarding detectedbehaviors of interest.

Vehicle 101 may also receive updated map or object data via network 820.For example, server 810 may provide vehicle 101 with new data relatingto object classifications and behavior model information. Computersystem 110, of FIG. 1 , may then be updated, and the new data may beused in controlling the vehicle autonomously, such as throughimplementation of flow diagram 700.

As these number and usage of these autonomous vehicles increases,various sensors and features may be incorporated into the environment toincrease the perception of the vehicle. For example, low-cost beacontransmitters may be placed on road signs, traffic signals, roads orother highway infrastructure components in order to improve thecomputer's ability to recognize these objects, their meaning, and state.Similarly, these features may also be used to provide additionalinformation to the vehicle and driver such as, whether the driver isapproaching a school or construction zone. In another example, magnets,RFID tags or other such items may be placed in the roadway to delineatethe location of lanes, to identify the ground speed vehicle, or increasethe accuracy of the computer's location determination of the vehicle.

Autonomous vehicles may also be controlled remotely. For example, if thedriver is asleep, the sensor data may be sent to a third party so thatvehicle may continue to have a responsive operator. While delay andlatency may make this type of telemetry driving difficult, it may forexample be used in emergency situations or where the vehicle has gottenitself stuck. The vehicle may send data and images to a central officeand allow a third party to remotely drive the vehicle for a short perioduntil the emergency has passed or the vehicle is no longer stuck.

As these and other variations and combinations of the features discussedabove can be utilized without departing from the invention as defined bythe claims, the foregoing description of exemplary embodiments should betaken by way of illustration rather than by way of limitation of theinvention as defined by the claims. It will also be understood that theprovision of examples of the invention (as well as clauses phrased as“such as,” “e.g.”, “including” and the like) should not be interpretedas limiting the invention to the specific examples; rather, the examplesare intended to illustrate only some of many possible aspects.

The invention claimed is:
 1. A method comprising: controlling, by one ormore computing devices, an autonomous vehicle in accordance with a firstcontrol strategy; receiving, by the one or more computing devices,sensor data indicating a detection of a first object; classifying, bythe one or more computing devices, the first object based on the sensordata; accessing, by the one or more computing devices, behavior databased on a classification of the first object, wherein the behavior dataidentifies potential actions of the first object that are to result in achange in control strategy, and wherein at least one of the potentialactions identified in the behavior data is the action of changing fromtraveling on a first road element to travelling on a second roadelement; determining, by the one or more computing devices, that thefirst object has performed an action identified in the behavior data;and based on the determination, altering the control strategy of theautonomous vehicle by the one or more computing devices.
 2. The methodof claim 1, wherein the first road element is a first lane of traffic ona roadway, and wherein the second road element is a second lane oftraffic on the roadway.
 3. The method of claim 1, wherein altering thecontrol strategy of the autonomous vehicle comprises positioning theautonomous vehicle relative to the first object in a predefined manner.4. The method of claim 1, wherein altering the control strategy of theautonomous vehicle comprises having the autonomous vehicle change fromtravelling in a first lance of traffic to travelling in a second lane oftraffic.
 5. The method of claim 1, wherein altering the control strategyof the autonomous vehicle comprises altering at least one of a position,heading, speed, and acceleration of the autonomous vehicle.
 6. Themethod of claim 1, wherein the first object is classified as a vehicle.7. The method of claim 6, wherein determining that the first object hasperformed an action identified in the behavior data further comprisesdetermining that the first object has changed from travelling on thefirst road element to travelling on the second road element.
 8. A methodcomprising: controlling, by one or more computing devices, an autonomousvehicle; receiving, by the one or more computing devices, sensor dataindicating a position of a first object external to the autonomousvehicle; classifying, by the one or more computing devices, the firstobject based on the sensor data; accessing, by the one or more computingdevices, map data having a plurality of road elements; comparing thesensor data with the map data; identifying, by the one or more computingdevices, that the first object is travelling on a first road elementfrom the plurality of road elements; determining, by the one or morecomputing devices, that based on the comparison of the sensor data withthe map data, the first object has travelled from the first road elementto a second road element; and altering, by the one or more computingdevices, at least one of a position, heading, speed, and acceleration ofthe autonomous vehicle based on the determination that the first objecthas travelled from the first road element to the second road element. 9.The method of claim 8, wherein the first road element is a first lane oftraffic on a roadway, and wherein the second road element is a secondlane of traffic on the roadway.
 10. The method of claim 8, whereinaltering at least one of the position, heading, speed, and accelerationof the autonomous vehicle comprises positioning the autonomous vehiclerelative to the first object in a predefined manner.
 11. The method ofclaim 8, further comprising: receiving, by the one or more computingdevices, a request to navigate between a first location and a secondlocation; and autonomously navigating, by the one or more computingdevices, the autonomous vehicle along a path between the first locationand a second location; and wherein altering at least one of a position,heading, and speed of the autonomous vehicle, occurs while theautonomous vehicle is travelling along the path.
 12. The method of claim8, further comprising: determining a relative position of the autonomousvehicle with each of the one or more nearby vehicles, and whereinassociating each of the one or more nearby vehicles with a road graphelement is based on the relative position.
 13. A system for controllingan autonomous vehicle, the system comprising: one or more sensors fordetecting a one or more vehicles in an autonomous vehicle'ssurroundings; and one or more processors configured to: control anautonomous vehicle in accordance with a first control strategy; receivesensor data indicating a detection of a first object; classify the firstobject based on the sensor data; access behavior data based on aclassification of the first object, wherein the behavior data identifiespotential actions of the first object that are to result in a change incontrol strategy, and wherein at least one of the potential actions ofthe first object is the first object changing from traveling on a firstroad element to travelling on a second road element; determine that thefirst object has performed an action identified in the behavior data;and alter the control strategy of the autonomous vehicle based on thedetermination.
 14. The system of claim 13, wherein the first roadelement is a first lane of traffic on a roadway, and wherein the secondroad element is a second lane of traffic on the roadway.
 15. The systemof claim 13, wherein altering the control strategy of the autonomousvehicle comprises positioning the autonomous vehicle relative to thefirst object in a predefined manner.
 16. The system of claim 13, whereinaltering the control strategy of the autonomous vehicle comprises havingthe autonomous vehicle change from travelling in a first lance oftraffic to travelling in a second lane of traffic.
 17. The system ofclaim 13, wherein altering the control strategy of the autonomousvehicle comprises altering at least one of a position, heading, speed,and acceleration of the autonomous vehicle.
 18. The system of claim 13,wherein the sensor data includes information relating to at least one ofa position, heading, speed, and acceleration of the first object. 19.The system of claim 18, wherein determining that the first object hasperformed an action identified in the behavior data further comprisesdetermining that the first object has changed from travelling on thefirst road element to travelling on the second road element.
 20. Thesystem of claim 13, wherein the one or more processors are furtherconfigured to: receive a request for navigation between a first locationand a second location; and autonomously navigate the autonomous vehiclealong a path between the first location and a second location; andwherein altering the control strategy occurs while the autonomousvehicle travels along the path.
 21. A method comprising: receiving, byone or more sensors of a vehicle configured to operate in a fullyautonomous mode, sensor data of an external environment of the vehicle;filtering, by one or more computing devices of the vehicle, the receivedsensor data to identify one or more actions of interest of an object inthe external environment, the filtering including filtering the receivedsensor data to only include instances where the object has performed anaction of interest; determining, by the one or more computing devices,whether the vehicle cannot currently operate in the fully autonomousmode based on the identified one or more actions of interest; and upondetermining by the one or more computing devices that the vehicle cannotcurrently operate in the fully autonomous mode, the one or morecomputing devices altering a control strategy; wherein altering thecontrol strategy includes either (i) delaying changing to the fullyautonomous mode, or (ii) ceding control of the vehicle to a remoteoperator.
 22. The method of claim 21, wherein the object is anothervehicle.
 23. The method of claim 21, wherein delaying changing to thefully autonomous mode includes the one or more computing devicesscanning the external environment to determine whether there are anyobstacles affecting an ability of the vehicle to avoid a collision. 24.The method of claim 21, wherein delaying changing to the fullyautonomous mode includes the one or more computing devices requiring adriver of the vehicle to control steering or accelerating beforeentering into the fully autonomous mode.
 25. The method of claim 21,wherein ceding control of the vehicle to the remote operator includessending the received sensor data to a remote party associated with theremote operator.
 26. The method of claim 25, further comprisingtransmitting data or imagery to a remote computing device in conjunctionwith ceding control to the remote operator.
 27. The method of claim 21,wherein the identified one or more actions of interest include anothervehicle changing lanes.
 28. The method of claim 21, wherein theidentified one or more actions of interest include another vehiclechanging its route.